Expressing Complex Mental States Through Facial Expressions

A face is capable of producing about twenty thousand different facial expressions [2]. Many researchers on Virtual Characters have selected a limited set of emotional facial expressions and defined them as basic emotions, which are universally recognized facial expressions. These basic emotions have been well studied since 1969 and employed in many applications [3]. However, real life communication usually entails more complicated emotions. For instance, communicative emotions like “convinced”, “persuaded” and “bored” are difficult to describe adequately with basic emotions. Our daily face-to-face interaction is already accompanied by more complex mental states, so an empathic animation system should support them. Compared to basic emotions, complex mental states are harder to model because they require knowledge of temporal changes in facial displays and head movements as opposed to a static snapshot of the facial expression. We address this by building animation models for complex emotions based on video clips of professional actors displaying these emotions. The first step of our work is to extract the facial and head movements from video clips. We have adopted the recognition framework proposed by [4] to recognize head and facial displays, which allows facial displays to be based on not just the current facial action, but also on a pre-determined number of previous facial actions. Using the framework in [4], a commercial face tracker is used to locate and track 24 landmarks on the face. These are then mapped into facial actions and displays using Hidden Markov Models (HMMs) trained on video clips from the Mind Reading DVD compiled by researchers at the University of Cambridge Psychology Department [1]. The likelihoods of the facial displays, computed by the HMMs from the input video sequences, are used to drive the animation of our virtual characters. Six head action displays(head tilt, head turn, head forward, head backward, head shake and head nod) and three three facial displays (lip pull, lip pucker, and brow raised) are extracted for every frame and applied on the Virtual Character. The first four of the HMM outputs for head movement were translated directly into rotation about a single axis whereas head shake and head nod were translated into periodic animations. The facial displays are applied on the Virtual Characters as morph targets. As described above complex mental states differ from basic emotions in that they cannot be effectively recognized from static facial expressions, only from